Methods and Apparatus Recognition of Start and/or Stop Portions of a Gesture Using Relative Coordinate System Boundaries

ABSTRACT

Described are apparatus and methods for reconstructing a gesture by aggregating various data from various sensors, including data for recognition of start and/or stop portions of the gesture using a detection of an intersection with a relative coordinate system boundary.

RELATED APPLICATION

This application claims priority to U.S. Provisional Patent ApplicationSer. No. 61/984,548 filed Apr. 25, 2014 and is a continuation-in-part ofU.S. application Ser. No. 14/591,878 filed Jan. 7, 2015, which claimpriority to U.S. Provisional Patent Application Ser. No. 61/924,682filed Jan. 7, 2014, each of which are expressly incorporated byreference herein.

FIELD OF THE ART

This disclosure relates to using the Human body as an Input mechanism,and, in particular, recognition of start and/or stop portions of agesture using relative coordinate system boundaries.

BACKGROUND

Many conventional gestural systems attempt to detect gestures thatresemble characters or words. Such conventional gestural systems,however, offer very poor recognition rates.

SUMMARY

Described are apparatus and methods for reconstructing a gesture in amanner that includes detecting when a user enters or leaves a boundaryspace associated with a relative coordinate system.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1(A) illustrates the skeletal rendering of the human with variousnodes, and the usage of many different sensors according to theembodiments.

FIG. 1(B)1 illustrates a system diagram according to an embodiment.

FIG. 1(B)2 illustrates a system diagram according to anotherembodiments.

FIG. 1(B)3 illustrates system diagram according to a further embodiment.

FIG. 2 illustrates that the system allows for the sensor 3 to be usedfor one gesture one pointing to a light (1) as shown in FIG. 2, andanother gesture when pointing at the computer (2) as shown.

FIGS. 3, 4, and 5 show embodiments for micro-gesture recognitionaccording to the embodiments.

FIG. 6 shows an illustration of micro-gestures detected within asubspace that has its own relative coordinate system.

FIG. 7 illustrates a 3D exterior view of a single ring sensor.

FIG. 8 illustrates a more detailed view of the ring sensor of FIG. 7.

FIG. 9 illustrates a computer sensor & receiver according to theembodiments.

FIG. 10 illustrates a flow chart of operation using the capacitive touchsensor and low power modes.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

Various devices such as computers, televisions, electronic devices andportable handheld devices can be controlled by input devices such as acomputer mouse or keyboard. Various sensors such as accelerometers,gyroscopes, compasses and cameras can be collectively used (all from asubstantially single point such as if disposed on a single ring; or frommultiple different locations) to estimate and/or derive a gesture thatis intended to have some significant meaning. These sensors dynamicallyprovide data for varying periods of time when located in the associatedspace for sensing, and preferably stop or go into a low power mode whennot in the associated space. When sensor data is unavailable, variouscalculations may be employed to reconstruct the skeletal structurewithout all the sensor data.

Various poses and gestures of the human skeleton over a period of timecan be aggregated to derive information that is interpreted (either atthe sensor or at the device) and communicated over wireless channelssuch as WiFi, Bluetooth or Infrared to control various devices such ascomputers, televisions, portable devices and other electronic devices,as described further herein and in the previously filed U.S. patentapplication Ser. No. 14/487,039 filed Sep. 14, 2014, which claimspriority to U.S. Provisional Application No. 61/877,933 filed Sep. 13,2013, and entitled “Methods and Apparatus for using the Human Body as anInput Device”, which are explicitly incorporated herein by reference.

Described are apparatus and methods for reconstructing a gesture byaggregating various data from various sensors, including data forrecognition of start and/or stop portions of the gesture using adetector that detects when a user enters or leaves a boundary spaceassociated with a relative coordinate system.

In a preferred embodiment, sensors such as MEMS or other sensors. andpreferably a plurality of them within a substantially single locationsuch as on a ring or some other wearable form factor are used, incombination with a capacitive touch sensor or a tactile switch orsensors used for recognition of start and/or stop portions of thegesture. MEMS sensors in particular provide the advantage of notrequiring a separate detector compared to conventional camera baseddepth sensors and don't have to be in the very restricted viewing areaof a conventional depth camera. A plurality of sensors can be used toobtain further information than would be possible with a single suchsensor, as described herein. When further used in combination withaccelerometers, gyroscopes, compasses, the data from the various sensorscan be fused and interpreted to allow for sensing of micro-gestures, asdescribed herein.

Such a single sensing device having multiple sensors can be integratedinto everyday objects such as clothing, jewelry and wearable deviceslike fitness monitors or augmented reality glasses in order to use ofthe human body as a real-time input device that can interact with amachine in its surroundings.

Processing of all the data generated to accurately detect the pose ofthe human body in real-time includes engineering desiderata of eventstream interpretation and device power management, as well as usage ofalgorithms to fuse the sensor data into coherent pose estimates. Thefiltering algorithms used are based on the locality of the sensor andfactor in the human anatomy and the joint angles of the bones thesensors are tracking. The fused data is then processed to extractmicro-gestures—small movements in the human body which could signal anintent, as described herein.

Gestures such as waving your arm from one side to another ormicro-gestures such as swiping your index finger from one side toanother are mapped to functions, such as changing channels on a TV oradvancing the song being played. More complex gestures, such asinteracting with the User Interface of a tablet computer are alsopossible using micro-gestural primitives to generate a more complexmacro intent that machines in the environment can understand. All ofthese gestures, however, must have start points and stop points, whichneed to be detected in some manner.

Thus an aspect of the system includes assembling a movement sequence(aka gesture) that could be used to indicate a command, for example,which has a start point and a stop point. Each gesture can also take ona different meaning depending on which device it is communicating with.Thus, pointing to a Television and moving your hand from one directionto another can imply changing the channel while a similar such gesturecould imply changing the light intensity when done pointing to a lightbulb, with each of the Television and the light bulb being separatesubspaces that are detected as such by an overall detector, for example.

Efficient power management strategy is also provided, such that thesensor device doesn't require a power on or power off switch. Thisinvolves determining the current state of gestural detection and furtherincludes the ability to turn off components such as the gesturaldetection unit, or various sensors to save power, and in particularusing a capacitive touch sensor or a tactile switch or a specificgesture or any combination of the three as described hereinafter toaccomplish certain of these power savings.

It is noted that the single sensing device is a battery-operated device,yet it does not have a power button. It does, however, have capacitivetouchpads and tactile switches as described, which can be programmed toactivate and/or de-activate the single sensing device, thereby ensuringthat the device is in use only when the user intends for it to be andkeeping it energy-efficient.

As described further herein, the detector can be located on the wearableinput platform or instead be remote from it, such as a camera. Iflocated on the wearable input platform, in association with locationdetection, a determination can be made that the user wants to startcommunicating with or no longer desires to stay communicating with thedevice that is within the current relative coordinate system.

This detector takes the guesswork out of the gesture acquisition engine,whereby it is not trying to interpret random gesture, unless expresslyinstructed to do so, via the output from the detector. Additionally, thedetector ensures that the single sensing device is energy efficient andactive only as needed for the duration of the interaction. Similarly,the gesture acquisition engine is not “always-on” and in use only whenneeded, thereby conserving energy.

These various aspects are shown in the diagrams attached. FIG. 1(A)illustrates the skeletal rendering of the human with various nodes, andthe usage of many different sensors: one on the glasses (1), another onthe belt (2), a third of a number of different sensors for fingers (3),one for the wrist (4) and one on an ankle bracelet or attached to thebottom of the pants worn (5). FIGS. 1(B)(1-3) shows a similar space andrendering, and points out specific sub-spaces associated with differentobjects; each of which can have their own relative coordinate system ifneeded. As shown, FIG. 1(B)1 illustrates a system diagram with a laptopas a third controllable device, which laptop includes an interactionplane and is labeled as Computer Sensor & Receiver to illustrate that itcan operate the software needed to fuse different sensor data together,as described elsewhere herein. FIG. 1(B)2 illustrates a system diagramwith a laptop as well, but this laptop shown only as having aninteraction plane, and operate upon a distributed system (such as withcloud processing). FIG. 1(B)3 illustrates an even simpler, which doesnot include the laptop at all within it. As is apparent, many differentcombinations are possible and within the contemplated scope herein.

As described above, the system allows for the sensor 3 to be used forone gesture one pointing to a light (1) as shown in FIG. 2, and anothergesture when pointing at the computer (2) as shown.

FIGS. 3, 4, and 5 show embodiments for micro-gesture recognition thatinclude usage of 1, 2 and 3 finger rings, respectively, as shown. Otherconfigurations are possible and within the intended scope herein.

FIG. 6 shows an illustration of micro-gestures that are detected withina subspace around a computer, which sub-space can have its own relativecoordinate system, rather than being based upon absolute coordinates,which subspace creates a boundary around the device, such as thecomputer shown. In addition to the sensor data in each ring that can beused as the detector, radio strength can also be used to as the detectordetect distance from a relative reference point, such as the screen ofthe computer. Additionally, the relative coordinate system can be basedon the part of the body to which the single sensing device is attached,with a ring on a finger having as a relative coordinate system theportion of the arm from the elbow to the wrist as one axis. If, forexample, the user's hand enters the boundary around the computer, thatcan be used to start the gesture recognition, and, similarly, if theuser's hand leaves the boundary around the computer, that can be used tostop the gesture recognition.

In an aspect relating to boundaries that are larger, such as pointing toa television screen or wall switch that is a larger distance away, asthe body turns toward or away from each such device, that turn cansimilarly be detected and used to start or stop the gesture recognition.

FIG. 7 illustrates a 3D exterior view of a single ring sensor, and FIG.8 illustrates that ring sensor in a more detailed view, with thesignificant electronic components identified, and which are connectedtogether electrically as a system using a processor, memory, software asdescribed herein, including other conventional components, forcontrolling the same. The processor controls the different sensors onthe ring device and is in charge of detecting activity in the varioussensors, fusing the data in them and sending such data (preferablyfused, but in other embodiments not) to other aggregators for furtherprocessing. While shown as a ring sensor, this combination of elementscan also be used for the other sensors shown in FIG. 1 —though othercombinations can also be used.

FIG. 9 illustrates a Computer Sensor & Receiver as shown in FIG. 1(B1).As illustrated in FIG. 9, included are a processor, memory and displaythat are used as is conventionally known. The processor controls thedifferent sensors on the various devices and can fuse the data fromdisparate devices that has been aggregated previously or not, and sendsuch data (preferably fused, but in other embodiments not) to otheraggregators for further processing as well as send control signals basedon the what has been detected to control devices such as the light ortelevision as shown in FIG. 1. I/O devices as known are also included,as well as what is labeled a Gesture Input/Output Device and anAggregator coupled thereto (which Aggregator may be part of the ComputerSensor and Receiver or could be located elsewhere, such as on a wristsensor as described above). The Aggregator can be implemented inhardware or software to process the various streams of data beingreceived from the various sensors. The Aggregator factors in location ofthe sensor (e.g: on the finger or wrist etc.) and calculates what datais relevant from this sensor. This is then passed on to the GestureInput/Output Device (which could also reside across a wireless link) tocontrol various computing devices.

The device can also possess a haptic actuator and associated circuitryto be able to provide a haptic feedback based on user engagement with acomputing device. The device can also support various forms of wirelessnetworking such as NFC, Bluetooth and/or WiFi to be able to interactwith various other devices in its surroundings.

Multiple sensors can interact with each other providing a stream ofindividually sensed data. For example a sensor worn on the ring cancommunicate with a wrist worn device or a smartphone in the pocket. Thisdata could then be aggregated on the smartphone or wrist worn devicefactoring in the human anatomy. This aggregation may factor in range ofmotion of the human skeletal joints, possible limitations in the speedhuman bones could move relative to each other, and the like. Thesefactors, when processed along with other factors such as compassreadings, accelerometer and gyroscope data, can produce very accuraterecognition of gestures that can be used to interact with variouscomputing devices nearby.

FIG. 10 illustrates a flowchart of the preferred operation using thedetector and low power modes, which is implemented in applicationsoftware loaded onto the memory and executed by the processor, inconjunction with the gesture input/output device, aggregator, andsensors. For understanding, operation of a single detector is explained,but it will readily be appreciated that the same operations are used formultiple detectors.

In operation, step 1010 of FIG. 10 shows the single sensor device beingin the “on” state and charged sufficiently for operation. If notcharged, then a separate charging station (not shown) can be used tocharge the device. After step 1010, step 1012 follows, with entry into alow power mode. In this low power mode, the minimum operations areperformed, and as many of the sensors and the like are put into a sleepstate in order to preserve power, with an auxiliary sensor, such as acapacitive touch sensor, being periodically awaken and scanned to see ifan event has occurred, in steps 1014. Further, other start events (suchas tactile or gestural input) can be programmed, and this is shown asstep 1016. In a preferred embodiment, the low power mode has a tactileonly input to wake up from deep sleep, and all other sensors are off, aswell as wireless transmission/reception. In both the medium and lowpower modes, wireless transmission/reception is preferably off

If in either of steps 1014 or 1016 a start signal is detected, thensteps 1018 follows, with the single sensor device entering the regularpower mode, such that full functionality is possible, though even withinfull mode power savings procedures can be put in place to conservepower.

One step as shown in the regular power mode is indicated as step 1020 inwhich gesture and other data are detected, until a stop signal isdetected, such as based upon the detection that the gesture input/outputdevice has left the relative coordinate system with which it had beencommunicating. Other full functionality steps can also occur, such asProcessing/transforming the gestures and other sensor data such asacceleration and orientation; transmitting the processed data over awireless medium to enable interaction with the smart device (TV, smartphone, tablet, etc.)

Steps 1022, 1024 and 1026 all follow, which are each detecting theexistence of the end of the gesture. Usage of the detector to detect aspecific stop gesture is shown in step 1022, whereas step 1024 showsthat an end of gesture can be detected based upon the gesture data(based on a pre-programmed, unique gesture). Step 1026 indicates that atime limit or other stop trigger (such as a tactile switch) can also beused to generate the stop signal at the end of a gesture. This detectorcan also be used in combination with a capacitive touch sensor asdescribed in U.S. patent application Ser. No. 14/591,878 filed Jan. 7,2015, which claims priority to U.S. Provisional Application No.61/924,682 filed Jan. 7, 2014, entitled “Methods and Apparatus forRecognition of Start and/or Stop Portions of A Gesture Using anAuxiliary Sensor”, expressly incorporated by reference above.

Upon detection of a stop signal in any of steps 1022, 1024 and 1026,step 1028 follows, and a medium power mode is preferably entered into,in which case the, for example, no further gesture data collection isperformed, the sensors are turned off, and processing of the gesturedata collected already is finished using time-keeping, so as to thenperform operations in accordance with such processed gesture data. Otherfunctions that may still occur in a medium power mode, that wouldpreferably not occur in a low power mode, are keeping all the sensorssensing (in standby) and waiting for some combination or one oftouch/gesture/tactile input for quick startup.

Following step 1028 is a step 1030, in which a preferably programmeddetermination of whether to then enter into the low power mode 1012, theregular power mode 1018, or stay in the medium power mode 1028.

Although the present inventions are described with respect to certainpreferred embodiments, modifications thereto will be apparent to thoseskilled in the art.

1. An apparatus capable of interacting with at least one controllabledevice based upon a pose of at least a portion of a human body, theapparatus comprising: one or more sensors that are sized for wearing onthe human body, each of the one or more sensors emitting sensor data andbeing packaged in an integrated mechanical assembly; and a detectionunit that operates upon the sensor data to determine the pose of atleast the portion of the human body and is capable of interacting withthe at least one controllable device, the detection unit including: amemory that stores at least one or more characteristics of human anatomythat are associated with the human body using at least a partialskeletal rendering of a human; and a detection processor, automaticallyoperating under software control, that inputs, aggregates and fuses thesensor data from each of the one or more sensors using the at least oneor more characteristics of human anatomy stored in the memory todetermine the pose of at least the portion of the human body based upona locality of said one or more sensors, wherein the detection processorbegins to input, aggregate and fuse the sensor data for gesturerecognition upon certain sensor data indicating that the at least one ofthe one or more sensors within the integrated mechanical assembly haveintersected into a relative coordinate system boundary to therebyinitiate a gesture start signal and wherein the detection processorceases to input the sensor data for gesture recognition upon receipt ofa gesture stop signal.
 2. The apparatus according to claim 1 wherein thegesture stop signal is generated by one of (a) other sensor dataindicating that the at least one of the one or more sensors within theintegrated mechanical assembly have intersected out of the relativecoordinate system boundary and (b) receipt of the gesture stop signalfrom an auxiliary sensor sized for wearing on the human body thatreceives a specific input based on one of a tactile switch andcapacitive touch input.
 3. The apparatus according to claims 2 whereinthe detection unit is also packaged in the integrated mechanicalassembly
 4. The apparatus according to claims 2 wherein the gesture stopsignal is generated by other sensor data indicating that the at leastone of the one or more sensors within the integrated mechanical assemblyhave intersected out of the relative coordinate system boundary.
 5. Theapparatus according to claim 2 wherein the relative coordinate systemboundary is related to a hand and arm, with a portion of the arm betweenthe elbow and wrist serving as one axis of the relative coordinatesystem.
 6. The apparatus according to claim 5 wherein the relativecoordinate system boundary is associated with an electronic devicedisposed remote from the integrated mechanical assembly.
 7. Theapparatus according to claim 6 wherein a distance between at least oneof the sensors on the integrated mechanical assembly and the electronicdevice is sensed to determine the sensor data indicating that the atleast one of the one or more sensors within the integrated mechanicalassembly have intersected into a relative coordinate system boundary tothereby initiate the gesture start signal.
 8. The apparatus according toclaim 2 wherein at least one of the gesture start signal and the gesturestop signal is triggered by a detected body turn toward or away from,respectively, an electronic device with which communication is capable.9. The apparatus according to claim 2 wherein the relative coordinatesystem boundary is based on a body part to which the integratedmechanical assembly will attach.
 10. The apparatus according to claim 1wherein the relative coordinate system boundary is related to a hand andarm, with a portion of the arm between the elbow and wrist serving asone axis of the relative coordinate system.
 11. The apparatus accordingto claim 1 wherein the relative coordinate system boundary is associatedwith an electronic device disposed remote from the integrated mechanicalassembly.
 12. The apparatus according to claim 1 wherein a distancebetween at least one of the sensors on the integrated mechanicalassembly and the electronic device is sensed to determine the sensordata indicating that the at least one of the one or more sensors withinthe integrated mechanical assembly have intersected into a relativecoordinate system boundary to thereby initiate the gesture start signal.13. The apparatus according to claim 1 wherein at least one of thegesture start signal and the gesture stop signal is triggered by adetected body turn toward or away from, respectively, an electronicdevice with which communication is capable.
 14. The apparatus accordingto claim 1 wherein the relative coordinate system boundary is based on abody part to which the integrated mechanical assembly will attach. 15.The apparatus according to claim 1 further including a timer that isused to create the gesture stop signal a predetermined period of timeafter generation of the gesture start signal.
 16. The apparatusaccording to claim 1 wherein the gesture start signal is further used tochange a power mode associated with the detection processor.
 17. Theapparatus according to claim 1 wherein the an auxiliary sensor sized forwearing on the human body that receives a first specific input based onone of a tactile switch and capacitive touch input changes the detectionprocessor into a regular power mode and the gesture stop signal causesthe one or more sensors to be turned off.
 18. The apparatus according toclaim 1, wherein the apparatus is further configured to interact with afirst device and a second device, and wherein an orientation of theapparatus relative to the first device causes the pose to be used tosignal the first device, and wherein orientation of the apparatusrelative to the second device causes the pose to be used to signal thesecond device.
 19. The apparatus according to claim 1 wherein thedetection processor receives further sensor data that represents a stoptime, and the gesture stop signal is generated therefrom.
 20. Theapparatus according to claim 1 wherein the detection processor receivesfurther sensor data that represents a start time, and the gesture startsignal is also generated therefrom.
 21. An method for interacting withat least one controllable device based upon a pose of at least a portionof a human body, the method comprising: sensing, using one or moresensors that are sized for wearing on the human body, sensor data fromeach of the one or more sensors, wherein the one or more sensors arepackaged in an integrated mechanical assembly; and determining the poseof at least the portion of the human body based upon the sensor data,under processor and software control, the step of determining operatingto : associate at least one or more characteristics of human anatomywith the human body using at least a partial skeletal rendering of ahuman; and automatically determine, under the processor and softwarecontrol the pose of at least the portion of the human body based upon alocality of said one or more sensors and the input from the auxiliarysensor, the step of automatically determining including inputting,aggregating and fusing the sensor data from each of the one or moresensors using the at least one or more characteristics of human anatomyto determine the pose, wherein the step of automatically determiningbegins to input, aggregate and fuse the sensor data for gesturerecognition upon certain sensor data indicating that the at least one ofthe one or more sensors within the integrated mechanical assembly haveintersected into a relative coordinate system boundary to therebyinitiate a gesture start signal and wherein the detection processorceases to input the sensor data for gesture recognition upon receipt ofa gesture stop signal.
 22. The method according to claim 11 furtherincluding the step of generating the gesture stop signal from anauxiliary sensor sized for wearing on the human body and packaged withinthe integrated mechanical assembly, the gesture stop signal is generatedby one of (a) other sensor data indicating that the at least one of theone or more sensors within the integrated mechanical assembly haveintersected out of the relative coordinate system boundary and (b)receipt of the gesture stop signal from the auxiliary sensor.
 23. Themethod according to claim 12 wherein the relative coordinate systemboundary is related to a hand and arm, with a portion of the arm betweenthe elbow and wrist serving as one axis of the relative coordinatesystem.
 24. The method according to claim 23 wherein the relativecoordinate system boundary is associated with an electronic devicedisposed remote from the integrated mechanical assembly.
 25. The methodaccording to claim 24 wherein a distance between at least one of thesensors on the integrated mechanical assembly and the electronic deviceis sensed to determine the sensor data indicating that the at least oneof the one or more sensors within the integrated mechanical assemblyhave intersected into a relative coordinate system boundary to therebyinitiate the gesture start signal.
 26. The method according to claim 22wherein at least one of the gesture start signal and the gesture stopsignal is triggered by a detected body turn toward or away from,respectively, an electronic device with which communication is capable.27. The method according to claim 22 wherein the relative coordinatesystem boundary is based on a body part to which the integratedmechanical assembly will attach.
 28. The method according to claim 21wherein the relative coordinate system boundary is related to a hand andarm, with a portion of the arm between the elbow and wrist serving asone axis of the relative coordinate system.
 29. The method according toclaim 21 wherein the relative coordinate system boundary is associatedwith an electronic device disposed remote from the integrated mechanicalassembly.
 30. The method according to claim 21 wherein a distancebetween at least one of the sensors on the integrated mechanicalassembly and the electronic device is sensed to determine the sensordata indicating that the at least one of the one or more sensors withinthe integrated mechanical assembly have intersected into a relativecoordinate system boundary to thereby initiate the gesture start signal.31. The method according to claim 21 wherein at least one of the gesturestart signal and the gesture stop signal is triggered by a detected bodyturn toward or away from, respectively, an electronic device with whichcommunication is capable.
 32. The method according to claim 21 whereinthe relative coordinate system boundary is based on a body part to whichthe integrated mechanical assembly will attach.
 33. The method accordingto claim 21 wherein a timer is used to create the gesture stop signal apredetermined period of time after generation of the gesture startsignal.
 34. The method according to claim 21 wherein the gesture startsignal is further used to change a power mode associated with theprocessor.
 35. The method according to claim 21 wherein the gesturestart signal changes the processor into a regular power mode and thegesture stop signal causes the one or more sensors to be turned off. 36.The method according to claim 21, wherein the integrated detection unitmechanical assembly interacts with a first device and a second device,and wherein an orientation of the an integrated detection unitmechanical assembly relative to the first device causes the pose to beused to signal the first device, and wherein orientation of theapparatus relative to the second device causes the pose to be used tosignal the second device.
 37. The method according to claim 21 whereinthe processor receives further sensor data that represents a stop time,and the gesture stop signal is generated therefrom.
 38. The methodaccording to claim 21 wherein the processor receives further sensor datathat represents a start time, and the gesture start signal is alsogenerated therefrom.